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ABSTRACT:  Modeling  and  simulation  (M&S)  plays  an  important  role  when  complex  human-system  notions  are 
being  proposed,  developed  and  tested  within  the  system  design  process.  National  Aeronautics  and  Space 
Administration  (NASA)  as  an  agency  uses  many  different  types  of  M&S  approaches  for  predicting  human-system 
interactions,  especially  when  it  is  early  in  the  development  phase  of  a  conceptual  design.  NASA  Ames  Research 
Center  possesses  a  number  of  M&S  capabilities  ranging  from  airflow,  flight  path  models,  aircraft  models,  scheduling 
models,  human  performance  models  (HPMs),  and  bioinformatics  models,  among  a  host  of  other  kinds  of  M&S 
capabilities  that  are  used  for  predicting  whether  the  proposed  designs  will  benefit  the  specific  mission  criteria.  The 
Man-Machine  Integration  Design  and  Analysis  System  (MIDAS)  is  a  NASA  ARC  HPM  software  tool  that  integrates 
many  models  of  human  behavior  with  environment  models,  equipment  models,  and  procedural  /  task  models.  The 
challenge  to  model  comprehensibility  is  heightened  as  the  number  of  models  that  are  integrated  and  the  requisite 
fidelity  of  the  procedural  sets  are  increased.  Model  transparency  is  needed  for  some  of  the  more  complex  HPMs  to 
maintain  comprehensibility  of  the  integrated  model  performance.  This  will  be  exemplified  in  a  recent  MIDAS  v5 
application  model  and  plans  for  future  model  refinements  will  be  presented. 


1.  Introduction 

Complex  system  integration  issues  require  that  the 
model  development  process  generally  follow  an 
iterative  design  philosophy  that  collaboratively 
leverages  empirical  human  data  (i.e.,  either  human  in 
the  loop,  HITL,  simulations  or  real-time 
measurements)  and  concurrently  feeds  information  to 
HITL  simulation  processes.  Many  organizations  are 
faced  with  the  goals  of  completing  research  as 
efficiently  as  possible  while  maintaining  acceptable 
levels  of  safety  to  successfully  complete  a  mission. 
NASA  is  no  exception.  Modeling  and  simulation 
techniques,  particularly  human  behavior  models,  play 
an  important  role  when  complex  human-system  notions 
are  being  proposed,  developed,  and  tested  across  many 
of  the  ten  NASA  centers.  For  instance,  NASA 
Johnston  Space  Center  (JSC)  utilizes  M&S  to  represent 
environments,  physical  structures  and  equipment 
components,  crew  stations,  planets  and  planetary 
motions,  gravitational  effects,  illumination,  human 
anthropometric  and  biomechanics,  among  a  host  of 
other  domains.  NASA  Ames  Research  Center  also 
possesses  a  number  of  M&S  capabilities  ranging  from 
airflow,  flight  path  models  (e.g..  Airspace  Concept 
Evaluation  System,  -  ACES),  aircraft  models, 
scheduling  models  (e.g.,  Core-XPRT,  Science  Planning 
InterFace  to  engineering  SPIFe),  human  performance 
models  (HPMs),  and  bioinformatics  models,  among 
many  other  kinds  of  M&S  capabilities.  One  of  the 
many  NASA  M&S  capabilities,  an  ARC-related  HPM 


capability  termed  the  Man-Machine  Integration  Design 
and  Analysis  System  (MIDAS)  is  highlighted  because 
of  its  relevance  to  the  field  of  human  behavior 
representation. 

1.1  Human  Performance  Models  (HPMs),  Concept 
Development  and  Testing 

Modeling  can  play  a  role  in  all  phases  of  the  concept 
development,  refinement,  and  deployment  process. 
Hybrids  of  continuous-control,  discrete-control  and 
critical  decision-making  models  represent  the 
‘internal  models  and  cognitive  function’  of  the 
human  operator  in  complex  control  systems,  and 
involve  a  coupling  among  humans  and  machines  in 
a  shifting  and  context  sensitive  environment.  These 
models,  known  as  HPMs,  have  arisen  as  viable 
research  options  due  to  decreases  in  computer  costs, 
increases  in  representative  results,  and  increases  in 
model  validity.  They  are  especially  valuable  because 
the  computational  predictions  can  be  generated  early  in 
the  design  phase  of  a  product,  system  or  technology  to 
formulate  procedures,  training  requirements,  and  to 
identify  system  vulnerabilities  and  where  potential 
human-system  errors  are  likely  to  arise.  The  model 
development  process  allows  the  designer  to  formally 
examine  many  aspects  of  human-system  performance 
with  new  technologies  to  explore  potential  risks 
brought  to  system  performance  by  the  human  operator 
(Gore  &  Smith,  2006).  Often  this  can  be  accomplished 
before  the  notional  technology  exists  for  human-in-the- 
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loop  (HITL)  testing  (Gore,  2000).  This  method 
possesses  cost  and  efficiency  advantages  over  waiting 
for  the  concept  to  be  fully  designed  and  used  in 
practice  (characteristic  of  HITL  tests).  Using  HPMs  in 
this  manner  is  advantageous  because  risks  to  the 
human  operator  and  costs  associated  with  system 
experimentation  are  greatly  reduced:  no  experimenters, 
no  subjects  and  no  testing  time  (Elkind  et  al.,  1989; 
Gore,  2000).  Hooey  and  Foyle  (2008)  outline  that 
HPMs  can  be  used  to  conduct  system  robustness 
testing  to  evaluate  the  system  from  the  standpoint  of 
potential  deviations  from  nominal  procedures  to 
determine  the  impact  on  the  performance  of  the  human 
and  the  system  (“what-if  ’  testing). 

1.2  The  Man-machine  Integration  Design  and 
Analysis  Systems  (MIDAS) 

MIDAS  is  a  dynamic,  integrated  human  performance 
modeling  and  simulation  environment  that  facilitates 
the  design,  visualization,  and  computational  evaluation 
of  complex  man-machine  system  concepts  in  simulated 
operational  environments  (Gore,  2008).  MIDAS 
combines  graphical  equipment  prototyping,  dynamic 
simulation,  and  HPMs  to  reduce  design  cycle  time, 
support  quantitative  predictions  of  human-system 
effectiveness,  and  improve  the  design  of  crew  stations 
and  their  associated  operating  procedures.  HPMs  like 
MIDAS  provide  a  flexible  and  economical  way  to 
manipulate  aspects  of  the  operator,  automation,  and 
task  environment  for  simulation  analyses  (Gore,  2008; 
Gore,  Hooey,  Foyle,  &  Scott-Nash,  2008;  Hooey  & 
Foyle,  2008). 

Gore  &  Smith  (2006)  outline  that  MIDAS  links  a 
virtual  human,  composed  of  a  physical  anthropometric 
character,  to  a  computational  cognitive  structure  that 
represents  human  capabilities  and  limitations.  The 
cognitive  component  is  composed  of  a  perceptual 
mechanism  (visual  and  auditory),  memory  (short  term, 
long  term-working,  and  long  term),  a  decision  maker 
and  a  response  selection  architectural  component.  The 
complex  interplay  among  bottom-up  and  top-down 
processes  enables  the  emergence  of  unforeseen,  and 
non-programmed  behaviors  (Gore  &  Smith,  2006). 
MIDAS  can  suggest  the  nature  of  pilot  errors,  and 
highlight  precursor  conditions  to  error  such  as  high 
levels  of  memory  demand,  mounting  time  pressure  and 
workload,  attentional  tunneling  or  distraction,  and 
deteriorating  situation  awareness  (SA). 


MIDAS  v  5  Task 
Network  and 
Behavior  Model 


MIDAS  v5 
Distributed  External 
Environment 
iMicrosaint  Sharp) 


and  Anthropometric 
(Jack™)  Model 


Figure  1.  MIDAS’  Environment,  Task,  and 
Anthropometric  Models. 

MIDAS  can  be  used  as  a  cognitive  modeling  tool  that 
allows  the  user  to  obtain  both  predictions  and 
quantitative  output  measures  of  human  performance, 
such  as  workload  and  SA  and  as  a  tool  for  analyzing 
the  effectiveness  of  crew  station  designs,  information 
display  concepts,  operator  roles  and  responsibilities 
from  a  human  factors  perspective  (Gore,  2008). 
MIDAS  has  proven  useful  for  identifying  general 
human-system  vulnerabilities  and  cross-domain  error 
classes  and  for  recommending  mitigation  strategies  and 
job  re-designs  to  account  for  the  vulnerable  areas,  or 
risks,  in  system  design  (Gore  &  Smith,  2006). 
Fundamental  design  issues  can  therefore  be  identified 
early  in  the  design  lifecycle,  prior  to  the  use  of 
hardware  simulators  and  HITL  experiments.  In  both 
cases,  MIDAS  provides  an  easy  to  use  and  cost 
effective  means  to  conduct  experiments  that  explore 
"what-if"  questions  about  domains  of  interest. 

1.3  The  MIDAS  User  Interface  Assists 
Comprehensibility 

MIDAS  v5  has  a  graphical  user  interface1  (GUI)  that 
does  not  require  advanced  programming  skills  to  use. 
The  GUI  brings  many  of  the  previously  embedded 
functions  to  the  surface  so  that  the  model  analyst  can 
observe  the  underlying  structure  as  well  as  the  model’s 
operation  as  it  is  run.  The  integrated  GUI  enables  the 
user  to  build  human  procedures  from  MIDAS  primitive 
tasks,  create  their  own  tasks,  incorporate  a  series  of 
nested  procedures,  change  the  SA  context  during  the 
simulation  and  manipulate  visual  and  auditory 


1  MIDAS  uses  Microsaint  Sharp  as  its  GUI  which  uses 
the  C-Sharp  programming  language 
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attributes  of  equipment  components.  The  MIDAS 
analyst  can  organize  the  human-system  interactions 
visually,  thereby  greatly  improving  the  model’s 
transparency.  Other  features  of  MIDAS  v5  include 
dynamic  visual  representations  of  the  simulation 
environment,  support  for  multiple  and  interacting 
human  operators,  distributed  simulation,  monte- 
carlo/stochastic  performance,  HPM  timelines,  task 
lists,  workload,  and  SA,  performance  influencing 
factors  (such  as  error  predictive  performance,  fatigue 
and  gravitational  effects  on  performance),  libraries  of 
basic  human  operator  procedures  (how-to  knowledge) 
and  geometries  for  building  scenarios  graphically  (that 
leverage  heavily  from  Siemens'  Jack  software). 

1.4  MIDAS  Approach  and  Land  Applications 

The  current  air  traffic  control  (ATC)  system  will  not  be 
able  to  manage  the  predicted  two  to  three  times  growth 
in  air  traffic  (JPDO,  2009).  The  Next  Generation  Air 
Transportation  System  (NextGen)  is  a  future  aviation 
concept  that  has  as  its  goals  to  significantly  increase 
the  capacity,  safety,  efficiency,  and  security  of  air 
transportation  operations  (JPDO,  2009). 

MIDAS  v5  has  been  applied  to  examine  a  NextGen 
approach  to  land  concept  termed  the  very  closely 
spaced  parallel  approach  (VCSPA).  In  order  to 
evaluate  this  concept,  two  MIDAS  v5  models  were 
generated.  The  first  was  a  current  day  Simultaneous 
Offset  Instrument  Approach  (SOIA)  model  that 
contained  the  current  day  procedures  and  the  second 
was  a  NextGen  VCSPA  model  that  contained 
predictive  displays  in  the  cockpit  and  a  modification  to 
the  roles  and  responsibilities  of  the  flight  crew  and 
ATC  modeled  operators.  This  simulation  involved  over 
500  tasks  and  culminated  in  a  verifiable  model  of 
approach  and  land  operations  (vetted  by  Subject  Matter 
Experts  -  SMEs).  The  SA  model  was  augmented 
within  MIDAS  to  represent  how  a  cockpit  crew  builds 
SA  of  traffic,  terrain,  and  weather  information  given 
the  accessibility  of  sources  of  information.  This  model 
effort  illustrated  the  “what-if’  simulation  capability 
within  MIDAS.  The  “what-if’  approach  was  completed 
when  MIDAS  was  exercised  with  one  set  of  displays 
and  procedure  sets  designed  to  represent  current  day 
operations  and  roles  followed  by  a  second  simulation 
with  an  alternate  set  of  displays  and  procedures 
encoded  to  represent  the  NextGen  displays  and 
expected  procedures.  The  model  underwent  an  iterative 
verification/validation  process  that  included 
examining:  (1)  the  task  sequences  and  the  performance 
of  the  model  as  it  executed;  (2)  the  visual  fixations, 
task  timings,  and  workload  relative  to  expected 
performance  given  the  inputs  to  the  model;  and  pilot 
performance  according  to  SME  evaluations. 

Model  comprehensibility  is  defined  as  understanding 


the  relationships  that  exist  among  the  models  being 
used  in  an  application,  the  performance  of  the  models 
in  the  application,  which  models  are  being  triggered  in 
the  model  architecture,  and  whether  the  model  is 
behaving  as  the  model  analyst  would  expect.  MIDAS 
v5’s  comprehensibility  was  greatly  improved  with  the 
transparent  model  architecture  (Gore,  2008).  The 
operation  of  this  complex  model  was  verified 
throughout  development  and  was  validated  according 
to  SME  evaluations.  The  verification  phase  of  the 
model  was  improved  given  the  visibility  into  the 
model’s  operations  at  any  given  point  in  simulation 
time  combined  with  the  cross  checking  of  the  jack 
visualization  and  the  simulation  runtime  data  that  was 
output.  The  comprehensibility  of  this  model  would  not 
have  been  possible  without  such  a  transparent 
architecture. 

This  MIDAS  v5  effort  lead  to  a  greater  awareness  of 
potential  parameters  that  should  be  included  in  system 
designs  and  enabled  the  research  program  to  visualize 
the  interactions  that  will  be  likely  in  future  NextGen 
operations.  It  is  anticipated  that  a  formal  validation 
approach  will  be  developed  and  applied  to  the  VCSPA 
model  in  an  upcoming  Federal  Aviation  Authority 
(FAA)  task.  This  FAA  task  will  require  model 
refinement  and  validation,  an  increased  number  of 
alternative  closely  spaced  operations  for  additional 
what-if  scenarios  including  alternative  pilot  roles  and 
responsibilities,  and  information  requirements. 

2.  Conclusion 

A  number  of  significant  challenges  exist  for  the  state  of 
the  art  in  HPMs,  two  of  which  will  now  be  highlighted. 

Transparency.  The  first  challenge  relates  to  model 
transparency.  Model  transparency  refers  to  the  ability 
to  comprehend  the  relationships  that  exist  among  the 
models  being  used  in  the  simulation,  the  performance 
of  the  models  in  the  simulation,  which  models  are 
triggering  in  the  model  architecture,  and  whether  the 
model  is  behaving  as  the  model  developer  would 
expect  (Gore,  2008).  Other  researchers  refer  to  this  as 
model  traceability,  model  behavior  visibility,  model 
verifiability,  and  model  interpretability  (Elkind  et  al., 
1989;  Napiersky,  Young,  &  Harper,  2004;  Gluck  & 
Pew,  2005;  Hooey  &  Foyle,  2008).  Transparency  in 
integrated  HPMs  is  needed  to  support  model 
verification,  validation,  and  credibility.  However, 
model  transparency  can  be  difficult  to  attain  because  of 
the  complex  interactions  that  can  exist  among  the 
cognitive,  physical,  environment  and  crew  station 
models,  and  because  the  cognitive  models  embedded 
within  integrated  HPMs  produce  behaviors  that  are  not 
directly  observable.  Three  types  of  transparency  that 
the  MIDAS  researchers  have  found  useful  to 
understand,  interpret,  and  increase  the  confidence  in 


16 


Proceedings  of  the  19th  Conference  on  Behavior  Representation  in  Modeling  and  Simulation,  Charleston,  SC,  21  -  24  March  2010 


the  complex  models’  output  include  transparency  of  the 
input,  transparency  of  the  integrated  architecture,  and 
transparency  of  the  output  (Gore,  Hooey,  Foyle,  & 
Scott-Nash,  2008).  This  paper  illustrates  how  the 
augmentation  to  the  MIDAS  GUI  has  improved  model 
transparency  that  has  led  to  better  model 
comprehensibility. 

Validation.  The  second  challenge  facing  the  HPM 
community  is  validation.  Validation  remains  a  very 
large  challenge  for  the  HPMs  community  because 
statistical  validation  is  oftentimes  seen  as  the  Holy 
Grail  for  determining  whether  a  model  is  suitable  but 
when  models  are  deemed  statistically  valid,  they  are 
less  generalizable,  and  less  re-usable  for  applications  in 
new  contexts.  This  places  the  field  of  modeling  into  the 
conundrum  of  making  models  that  are  statistically  valid 
(correlation,  r=.99)  but  that  lack  the  ability  to 
generalize  to  other  tasks  or  scenarios.  When  the 
generalizability  of  the  model  is  limited,  then  its  value 
as  a  cost-effective  approach  to  predict  complex  human- 
system  interactions  is  reduced. 

Validation  is  further  challenged  when  modeling  future 
technology  concepts  where  no  or  little  HITL  data  exists 
upon  which  to  statistically  validate  a  model  (as  in  the 
NextGen  aviation  systems  or  concepts  being  designed 
for  the  Space  program).  It  is  argued  that  our  definition 
of  model  validation  must  be  expanded  beyond  that  of 
statistical  results  validation  to  be  more  representative 
of  a  model  develop-model  verify-model  manipulate  - 
model  validate  iterative  process. 
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